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(g) A display system for visualization of body structures during medical procedures. 



@ An interactive display system superimposes 
radiological images on a semi-transparent 
screen through which a surgeon views a patient 
during a medical procedure. The superimposed 
image is derived from image data obtained with 
an imaging system. The radiological image is 
registered with the surgeon's view of the patient 
and displayed in real-time during a medical 
procedure. This allows the surgeon to view 
internal and external structures and the relation 
between them simultaneously, and adjust the 
procedure accordingly. A second embodiment 
employs stereoscopic viewing methods to pro- 
vide three-dimensional representations of the 
radiological images superimposed on the 
semi-transparent screen through which the sur- 
geon views the patient. 
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monitored in real-time. This information is then used 
to adapt the computer generated display on the semi- 
transparent screen so that the visual and computer 
generated images consistently coincide. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The features of the invention believed to be novel 
are set forth with particularity in the appended claims. 
The invention itself, however, both as to organization 10 
and method of operation, together with further ob- 
jects and advantages thereof, may best be under- 
stood by reference to the following description taken 
in conjunction with the accompanying drawing in 
which: is 

Fig. 1 is a simplified block diagram of a first em- 
bodiment of a medical display apparatus according to 
the present invention. 

Fig. 2 is a simplified block diagram of a second 
embodiment of a medical display apparatus accord- 20 
ing to the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

In Fig. 1, a patient 1 on which a medical proce- 25 
dure such as surgery is to be performed, is scanned 
by a medical imaging apparatus 10 which may be a 
magnetic resonance (MR) imaging apparatus, a com- 
puted axial tomography (CAT) apparatus, a positron 
emission tomography (PET) or similar imaging device 30 
capable of creating multi-dimensional volumetric 
data such as 3-dimensional (3-D) data, from internal 
structures of the patient After imaging, apparatus 1 0 
provides the volumetric data to a model workstation 
1 00. Once the volumetric data has been provided to 35 
model workstation 100, further need for imaging ap- 
paratus 10 imaging apparatus is no longer required. 
This is important since some medical procedures 
need not be performed with the patient situated within 
the confines of an imaging apparatus, which can be 40 
constricting in the case of MR imaging. In alternative 
embodiments, imaging apparatus 10 may be interac- 
tively employed during the medical procedure. Model 
workstation 100 stores the volumetric data and cre- 
ates computer generated models from the data cape- 45 
ble of being scaled, rotated and otherwise manipulat- 
ed, without the further need for imaging apparatus 
10. 

An operator 350, such as a physician or medical 
assistant, monitors patient 1. A semi-transparent so 
screen 250 is interposed between patient 1 and op- 
erator 350. A tracking device 50 which monitors and 
tracks operator 350, semi-transparent screen 250 
and patient 1 determines a relative roll a, pitch 6, and 
yaw $ orientation between operator 350, semi- 55 
transparent screen 250 and subject 1. Tracking de- 
vice 50 may be a 6-degrees of freedom tracking de- 
vice as described "The Flock of Birds" Installation and 



Operation Guide, Ascension Technology Corporation, 
July 5. 1992. in the Introduction pp. 1-3, Appendix 1, 
p. 89 and Appendix 3, p. 93. Tracking device 50 also 
determines a location (in Cartesian coordinates) of 
operator 350 and semi-transparent screen 250 with 
relation to patient 1. Patient 1 is assumed to be at the 
origin of the Cartesian coordinate system (x.y.z) = 
(0,0,0), therefore all distances relative to the patient 
are simply the (x,y,z) location. The location and ori- 
entation are interactively provided to model worksta- 
tion 100 by tracking device 50. The location and ori- 
entation may also be provided manually to model 
workstation(s) in different embodiments. 

Model workstation 100 processes the 3D volu- 
metric data it receives and creates selected render- 
ings of the data. One rendering method determines 
surfaces between differing types of tissue. Connec- 
tivity of similar types of tissue adjacent to one another 
is then determined. Differentiation of tissue types 
based on the nature of the signal in the three- 
dimensional image data is known as segmentation. 
When the 3-D volumetric data has been segmented 
into internal structures, each internal structure may 
be treated as a separate solid object by model work- 
station 100. The model workstation has the capability 
of selectively displaying desired internal structures, 
color coding structures and severing, rotating and 
translating internal structures in order to manipulate 
these images in a desired manner to provide visuali- 
zation to an operator working model workstation 100. 

An alternative rendering method generates two- 
dimensional projections of selected features within 
the three-dimensional data set is described in Euro- 
pean Patent Application 0506302 "Projection Meth- 
ods for Producing Two-Dimensionai Images from 
Three-Dimensional Data". For example, two-dimen- 
sional projection angiograms can be extracted from a 
three-dimensional phase contrast or time-of-f light 
magnetic resonance angiogram. Several projection 
algorithms are possible. These indude the detection 
of the maximum pixel intensity along a selected pro- 
jection ray through the three-dimensional data, deter- 
mination of the average pixel intensity of a selected 
projection ray and the determination of the standard 
deviation of all pixels along a selected projection ray. 

Model workstation 100 receives input data from 
a model cut plane input device 40 and a workstation 
view input device 60 to select the method of display- 
ing internal structures of patient 1 . Model cut plane in- 
put device 40 and a workstation view input device 60 
may be a computer pointing device such as a mouse 
or trackball, or any input device which indicates 
planes in which to cut the images and a viewing angle 
and scale. Tracking device 50 provides relative orien- 
tation data between operator 350 and patient 1 and 
scaling, which allows model workstation 100 to syn- 
thesize an interactive computer generated image of 
internal structures of patient 1 and have it coincide 
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segmented into what resembles a solid object, it may 
be manipulated as a solid object. In the case of struc- 
tures of a patient, a surgeon may acquire data from 
the patient by medical imaging, then plan surgery by 
manipulating the models to plan a desired result be- 5 
fore surgery. This is common in complex reconstruc- 
tive surgery. Once the plan is determined, it may be 
stored and played back during surgery. The images 
of internal structures are interactively oriented and 
scaled to coincide with the actual patient io 

A user employs as a model cut plane input device, 
a workstation view input device (40. 60 of Figs. 1 and 
2, respectively) to select planes in which to cut the 
structures in the model, to select a three-dimensional 
orientation of the model, and to select screen cut is 
planes which define a workstation viewing region. 
The model workstation can incorporate a dipping cir- 
cuit to determine points within the model cut planes, 
a rotation circuit to rotate points and normal vectors, 
a segmentation processor, and a shading circuit 20 
which determines shading based upon the orientation 
of the normal vector at each point In addition a screen 
clipping circuit can be used to determine points within 
a region defined by the screen cut planes. The model 
workstation also can include a display circuit to create 25 
video signals which, when propagated to a suitable 
display device, generate images of multiple surfaces 
that are within the desired display region and the 
screen cut planes. 

In radical surgery such as ablative surgery, or 30 
massive trauma cases, there is little structure which 
remains to correctly determine what a normal anato- 
my should be. In these cases, an additional model 
workstation may have a model of normal structures 
stored which may be mixed with the other images be- 35 
ing displayed to act as a guide in reconstructive sur- 
gery. This may be implemented by additional work- 
stations or model manipulation boards. 

In the present embodiments of the invention, 
semi-transparent screen 250 is interposed between 40 
operator 350 and patient 1. Screen 250 can be con- 
structed with a liquid crystal display or it can be com- 
prised of a partially silvered mirror reflecting an image 
from a video monitor. Semi-transparent display 250 
can be constructed as a relatively large device having 45 
dimensions approximately equal to that of the region 
of interest of patient 1, or alternatively it can be of 
small dimension and placed relatively close to the op- 
erator's eyes, perhaps incorporated into headgear or 
eyewear. 50 

While several presently preferred embodiments 
of the novel visualization system have been descri- 
bed in detail herein, many modifications and varia- 
tions will now become apparent to those skilled in the 
art. It is, therefore, to be understood that the append- 55 
ed claims are intended to cover all such modifications 
and variations 



Claims 

1. A real-time medical apparatus for displaying to an 
operator interactive images of internal structures 
of a patient coinciding with a view by the operator 
of the patient comprising: 

a) a medical imaging system for obtaining 
multi-dimensional imaging data of internal 
structures of said patient; 

b) a workstation for creating images from the 
imaging data of internal structures of said pa- 
tient viewed from a location (x,y,z) and orien- 
tation (0,4,6) which coincide with said view of 
the patient by said operator; 

c) a semi-transparent screen adapted for al- 
lowing said operator to view said patient 
through the semi-transparent screen and 
adapted for displaying the images of internal 
structures from the workstation with a desired 
degree of transparency to create the illusion 
of internal structures superimposed upon 
said patient 

2. The real-time medical apparatus of claim 1 fur- 
ther comprising a tracking device adapted to 
measure the location (x,y,z) and orientation 
(a,+,6) of said operator with respect to said pa- 
tient for repeatedly providing these measure- 
ments to the workstation. 

3. The real-time medical apparatus of claim 1 
wherein the workstation further comprises surgi- 
cal planning means adapted to create models of 
internal structures of said patient, interactively 
manipulate the models to result in modifications 
to internal structures, store the models and mod- 
ifications and display the models and modifica- 
tions viewed from a location and orientation to co- 
incide with the view of the patient by the operator. 

4. The real-time medical apparatus of claim 1 
wherein the workstation incorporates normal 
anatomical models of internal structures, and is 
adapted to display the models viewed from a lo- 
cation and orientation to coincide with the view of 
the patient by the operator, such that said work- 
station can act as a guide in reconstructive sur- 
gery. 

5. A real-time medical apparatus fordisplaying to an 
operator interactive three-dimensional (3D) inter- 
nal and external images of a patient comprising: 

a) a medical imaging system for obtaining 
three-dimensional (3D) imaging data of inter- 
nal structures of said patient 

b) tracking means for measuring locations 
(X1.V1.Z1) (X2.V2.Z2) and orientation angles 
(ai.4i.0d (a2.42.O2) of each eye of said opera- 
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